The Simpson’s Paradox in the Offline Evaluation of Recommendation Systems

Recommendation systems are often evaluated based on user’s interactions that were collected from an existing, already deployed recommendation system. In this situation, users only provide feedback the exposed items and they may not leave other since have been to them by As a result, dataset is used evaluate new model influenced system, as form of closed loop feedback. article, we show typical offline evaluation recommender suffers so-called Simpson’s paradox. paradox name given phenomenon observed when significant trend appears in several different sub-populations observational data but disappears or even reversed these combined together. Our in-depth experiments stratified sampling reveal very small minority frequently system plays confounding factor systems. addition, propose novel methodology takes into account confounder, i.e., system’s characteristics. Using relative comparison many models systems, Kendall rank correlation coefficient, our proposed exhibits statistically improvements 14% 40% examined open datasets (Yahoo! Coat), respectively, reflecting true ranking with (randomised) standard evaluation.